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About Success Boston 


Success Boston is Boston’s citywide college completion initiative. Together, the Boston Foundation, the Boston 
Public Schools (BPS), the City of Boston, the Boston Private Industry Council, 37 area institutions of higher 
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to double the college completion rate for students from the BPS. Success Boston was launched in 2008 in response 
to a longitudinal study by Northeastern University’s Center for Labor Market Studies, which showed that only 
35% of those BPS graduates who had enrolled in college ever completed a postsecondary certificate, Associate’s 
or Bachelor’s degree within seven years of graduation from high school. Together, the partner organizations 
implemented a three part strategy: getting ready, getting in, and getting through—to ensure Boston’s young 
people are prepared to meet the challenges of higher education and achieve a degree that will allow them to thrive 
in the workplace. Recently, Success Boston has expanded its mission to include “getting connected” to the labor 
market upon graduation from college. In 2014, the Boston Foundation received a grant from the Corporation 
for National and Community Service to expand this effort. This $6 million Social Innovation Fund award gives 
the Foundation the resources necessary to expand Success Boston’s transition coaching model from serving 300 to 


1,000 students from each of the Boston Public Schools classes of 2015, 2016 and 2017. 


About Abt Associates 


Founded in Cambridge, Massachusetts in 1965, Abt provides applied research and consulting services to 
government agencies, philanthropic, nonprofit, and commercial organizations around the world. Abt’s mission is 
to improve the quality of life and economic well-being of people worldwide. It applies its exceptional subject matter 
expertise, outstanding technical capabilities in applied research, and strategic planning to help local, national and 


international clients make better decisions and deliver better services. 
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Introduction 


Today, earning a college degree is seen as crucial 
for future well-being. College graduates earn more, 
are less likely to suffer job losses in a recession, 
and are projected to have superior long-term labor 
market prospects.’ Nationally, more than three of 
ten jobs already require postsecondary education" 
and more than six of ten current jobs are filled 

by candidates with postsecondary education." 
These figures reflect the competitive advantage of 
postsecondary education: even when a job does 
not explicitly require a degree, a candidate with 

a degree will tend to be hired over an equally 
qualified candidate without one. By 2020, over 70 
percent of Massachusetts jobs are projected to be 
filled by workers with postsecondary credentials,” 
a proportion higher than the state’s likely supply of 
college graduates, creating additional competitive 
pressure on Massachusetts residents in the labor 
market. In Boston, the six-year college graduation 
rate for the city’s 2009 public high school graduates 
who enrolled in college was 51 percent.” This rate 
improves upon the 39 percent seven-year rate 

for 2000 graduates, yet is not sufficient to meet 

the predicted demand for a college-educated 


workforce.! 


Students from low-income backgrounds and 


racial/ethnic minority groups may face social, 


Success Boston Coaching: 
2013-14 and 2014-15 


During the years covered in this study, 
seven local nonprofit organizations provided 
one-on-one coaching to students for their 
first two years in college. The organizations 
included: American Student Assistance, 
Boston Private Industry Council, Bottom 
Line, Freedom House, Hyde Square Task 
Force, Sociedad Latina, and West End 
House. Another nonprofit organization, 
uAspire, provided financial aid assistance 
and advising to Success Boston students, as 
well as training for Success Boston coaches. 


College/university partners included: 
Benjamin Franklin Institute of Technology, 
Bridgewater State University, Bunker Hill 
Community College, Massachusetts Bay 
Community College, Northeastern University, 
Roxbury Community College, Salem State 
University, Suffolk University, and University 
of Massachusetts Boston 


Prior research found that more coached 
students persisted in college than 
non-coached students’ 


academic, and logistical barriers to succeeding in college, may lack the supports needed to overcome 


barriers,’ and may struggle with managing key deadlines, such as financial aid and course registration 


deadlines’ One-on-one coaching from experienced counselors when students are completing their senior 


year in high school and beginning college can help them succeed.""# 


* A 2008 report, Getting to the Finish Line: College Enrollment and Graduation, A Seven-year Postsecondary Longitudinal Study of 
the Boston Public Schools Class of 2000 Graduates, found that 64% of nearly 3,000 BPS Class of 2000 graduates enrolled ina 


postsecondary institution within the first seven years of high school graduation, yet only 35.5% of college enrollees had 


earned a certificate, a two-year degree, or a four-year degree (Sum et al. 2008). That figure was later revised to 39%. 
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The Success Boston initiative was launched in 2008 specifically to improve Boston public high school 
graduates’ access to and success in college, particularly for members of groups traditionally underrep- 
resented in college degree attainment. Higher college graduation rates, in turn, are expected to increase 
students’ access to employment in such local industries as technology, financial services, higher educa- 


tion, and medical sectors that routinely require advanced training. 


The Success Boston initiative unites the Boston Foundation, City of Boston, Boston Public Schools (BPS), 
University of Massachusetts Boston, Bunker Hill Community College, other regional colleges and 
universities, uAspire, the Boston Private Industry Council, and other local nonprofit organizations in 

a concerted effort to improve college completion rates for Boston public school graduates. By offering 
academic programming and college advising activities in high school, one-on-one coaching support as 
students move into and through the first two years of college, and close collaboration with local higher 
education institutions, Success Boston aims to help BPS graduates earn degrees and enter the local work- 
force successfully. The Boston Foundation is the convening backbone organization of the Success Boston 
initiative; it funds and supports the nonprofit organizations that provide one of the core programs within 


the overall initiative: one-on-one transition coaching, or Success Boston Coaching (SBC). 
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Success Boston Coaching 


The Boston Foundation funded seven nonprofit organizations to provide coaching for 750+ first- and 
second-year students enrolled in over 30 nearby colleges during the 2013-14 and 2014-15 years. These 
seven organizations already had programs for BPS students in place. SBC focuses purposefully on easing 
the transition from high school to college, and ultimately, increasing college completion. Nonprofit 
organizations, partner colleges, and the Success Boston network collaborate to provide transition coaching 
to support students on the path to college graduation (see Exhibit 1). 


EXHIBIT 1 
Success Boston Coaching model 


SUPPORT 


IDENTIFY 
© Students and coaches @) 
connect in-person Coaches provide 


DAU ale on and off campus, ongoing, one-on-one GRADUATE 
recruit Boston hig! via text, email, support to students 


school graduates in or phone on academic, financial, 


transition to college career, and 
personal topics 


IN Yolaleyxeniimedgeelalyzctalolars 
Eaves (oXer1| 
colleges partner 
to coordinate coaching 
on campus 


CONNECT 


Coaches refer students 
to resources and 
support services at their 
colleges 


PARTNER 


SUCCESS BOSTON COACHING NETWORK 


SBC coaches support students throughout the academic year, offering on-demand guidance to help 


REFER 


prepare students to navigate, increasingly independently, through to college graduation.” They provide 
support on life skills, study skills, help-seeking strategies, and academic skills; they help students develop 
meaningful relationships, clarify goals, access networks, understand college culture, and make college life 
feasible; and they provide job and career mentoring. Partner colleges communicate with coaches and help 


? For a detailed description of how SBC was implemented in the 2014-15 academic year, see the report Degrees of Coaching: 
Success Boston’s Transition Coaching Model (Linkow et al. 2015). 
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coordinate coaching services on their campuses. The SBC network, which the Boston Foundation oversees, 
facilitates communication across organizations; it also provides coaches access to specialized training 
about financial aid from uAspire, a national nonprofit organization, as well as access to training on other 


topics. 


Since the Success Boston initiative’s start in 2008, SBC has been offered to eight successive cohorts of 
Boston public high school graduates, starting with the class of 2009.? An earlier evaluation demonstrated 
that coaching substantially increased student persistence in college for students from the BPS graduating 
class of 2009.x Findings presented in this brief build on the work of Andrew Sum and his colleagues at 

the former Northeastern University Center for Labor Market Studies, by including additional cohorts of 
students (classes of 2013 and 2014), by examining academic achievement and FAFSA renewal outcomes in 
addition to persistence, and by exploring the relationships between implementation and student outcomes. 


> Two later cohorts, college entrants in fall 2015 and fall 2016, participate in an expansion of Success Boston Coaching called 
Boston Coaching for Completion (BosC4C). Implementation of BosC4C is examined in the 2015-16 Implementation Report 
(Linkow et al. 2017, 2015-16 Implementation Report: Boston Coaching for Completion (BosC4C). Boston, MA: The Boston 
Foundation.). 
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This brief summarizes results from the second of three reports examining the classes of 2013 and 2014.* 


Study Design 


Using data from BPS, the Massachusetts Department of Elementary and Secondary Education (MA 


DESE), nine partner colleges, and the SBC program, the full report assesses whether and how coaching 
affects students’ short-term college outcomes.” The report also examines variation in program impacts 
according to selected student characteristics and features of coaching. It focuses on two specific cohorts 
of students who participated in SBC via seven nonprofit organizations: those who graduated from BPS in 


Key Design Features 


This study employs the strongest design possible to support causal claims about the effects of SBC on 


students. It: 


Follows students who graduated from Boston and Boston-area public high schools in 2013 and 2014 


for two and three years, respectively 


Uses a quasi-experimental design that matches SBC students to similar students who did not 


participate in coaching to create a comparison group of students as statistically similar as possible on 


observable characteristics to the treatment students before participation in SBC 


Matches on two features guided by best practice in social science methods: /oca/ treatment and 
comparison students are from similar high school environments and identical college settings, and 
also focatreatment and comparison students are carefully matched to be statistically equivalent 
on baseline characteristics (e.g., gender, race/ethnicity, high school academic achievement, 
socioeconomic status) believed to predict both selection into the SBC program and outcomes of 


interest’ 


Constructs two groups (one group of treatment and one of comparison students), who are similar 
on observable characteristics at the beginning of the study, or baseline, to rule out the possibility 
that baseline characteristics themselves account for any observed differences between SBC and 


non-coached students 


Uses data from BPS, MA DESE, National Student Clearinghouse, colleges in which BPS students 
enrolled, and the SBC program database 


Measures three outcome domains: persistence, achievement, and financial aid 


Conducts exploratory analyses to assess whether and how observed impacts of SBC vary as a 
function of particular student characteristics or features of coaching 
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2013 and 2014, who have been out of high school for three and two years, respectively, and who entered 
college in the falls of 2013 and 2014. Because the study uses a rigorous quasi-experimental design, it 
provides causal evidence that observed differences in outcomes between coached and similar non-coached 
students are due to participation in SBC. 


Although quasi-experimental designs that use local and focal matching can generate causal impact esti- 
mates, and therefore can approximate results from randomized control trials,’ there could be confounding 
factors not fully accounted for in this design. The matched comparison group does not eliminate bias 

that a random assignment process generally does, to the extent that observed impacts reflect, in part, the 
influence of unmeasured individual student characteristics on both participation in coaching and college 
outcomes. Nonetheless, the SBC recruitment processes and the sample construction each individually and 
taken together help reduce the likelihood of confounding factors. Students are recruited in SBC through 
multiple avenues: referrals from high school guidance counselors and other community organizations; 
nonprofit organizations’ middle school and high school programming pipelines; word of mouth; and 
sometimes from college referrals. The combination of recruitment pathways means that successive cohorts 
of SBC students likely have differential levels of motivation, attachments to the nonprofit organizations, 
and willingness to seek support. Also, the study sample uses the most inclusive definition of what it 
means to be an SBC student: all students initially recruited into the SBC program and who appeared in the 
program database—including those few without a single recorded interaction with a coach—are consid- 
ered SBC students. Therefore, the analytic sample includes students whose engagement and participation 
ranges from minimal to intensive. 


The analyses were limited to those students who met the following three specific criteria; they had to: 
® enroll in college in the fall after high school graduation 


® enroll in a college in which at least one SBC student and at least one potential comparison student 
were enrolled in that given year 


® have no missing information on key baseline characteristics (free and reduced-price lunch status 
and high school GPA) used in the matching process 


After applying these criteria, the evaluation sample included up to 722 SBC students and 2,261 non-SBC 
students; the specific number varies by outcome due to data availability. 


The outcomes for this evaluation fall into three domains: persistence, academic achievement, and financial 
aid (see Exhibit 2). Domains with multiple outcomes, persistence and achievement, distinguish between 
primary and exploratory outcomes. Primary outcomes are those most closely related to the theory of change, 
which hypothesizes that the elements of one-on-one coaching that together address logistical, academic, 
financial, and emotional support topics can improve traditionally underrepresented college students’ 


rs 


The first report, produced in November 2015, examined how nonprofit partner organizations implemented coaching for 
2013 and 2014 high school graduates (Linkow et al. 2015). The third and last report, due out in 2020, will explore long- 
term outcomes. 


a 


Methodological research using within-study comparisons (or design replication studies) inform best quasi-experimental 
design practices by comparing impact estimates based on comparison groups constructed by quasi-experimental designs 
versus those based on experimental designs to identify specific features of quasi-experimental designs that come closest to 
experimental results. See the full report for more detail. 
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persistence and completion rates. Explor- EXHIBIT 2 


atory outcomes are also informed by the Primary outcome measures 
theory of change, as they may help explain 


why or why not impacts are detected on DOMAIN PRIMARY OUTCOME MEASURES 


the primary outcomes, the most important Persistence into the Second Year 
ich i ; 5 Persistence 

ot which is soles completion (e.g., full Mae en a re 

time status is not an outcome in and of 

‘ pes : Cumulative GPA 

itself but it is useful to examine because paievement 

it is related to persistence and, ultimately, Good Academic Standing 

completion). This brief focuses on the five Financial Aid FAFSA Renewal 


primary outcomes (see the full report for a 
comprehensive discussion of exploratory 
outcomes).° 


The study purposively limited analysis of outcomes to the two-year period following high school gradua- 
tion so as to standardize the amount of time elapsed for both 2013 and 2014 high school graduating classes 
(which means those results can be pooled together), with one exception: persistence into the third year is 
measured about two and a half years after high school graduation and is only available for the class of 2014. 


The study also considered whether any 
observed impacts vary as a function 


Definitions of Selected Moderators 


of particular student characteristics or 


Underrepresented minority: categorized as traditionally famines of coaching by exploring now 


underrepresented in postsecondary education 
for Black, Hispanic, Native American or Other/ 


student characteristics and features of 
coaching are related to the impacts. For 


example, SBC may have an impact on 
Multiracial students and categorized as not traditionally 


underrepresented for White and Asian/Pacific Islander 
students. 


students traditionally underrepresented in 
postsecondary education (and not on those 
traditionally represented), or a differential 


Frequency: the number of interactions across all modes impact for students who have experienced 


(in-person, text, email, phone, and social media) 
between students and their coaches across the same 
two-year time span as described above, with high 
frequency measured as more than 9 meetings per year 
and low frequency measured as 9 or fewer interactions. 


more coaching interactions (rather than 
fewer) during the academic year. In 
statistical terms, such characteristics and 
features may moderate program impacts” 
These analyses are exploratory because: 
(1) they investigate impacts on subsets of 


the sample, and because the overall study 


® Exploratory outcomes include: continuous enrollment, full-time status, semesters enrolled in non-credit-bearing courses, 
and credit accumulation. 


” Two subgroup categories were created for each moderator. For example, race/ethnicity was structured into two 
categories (underrepresented and not underrepresented) rather than one category for each of several racial/ethnic groups 
(Black, Hispanic, White, Asian, Native American, and Other/Multiracial), chiefly to ensure that the subgroup sizes would 
have sufficient statistical power (or be large enough) to detect an impact of coaching, if it exists. This categorization also 
helps to generate easily interpretable comparisons between subgroups. 
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sample has been divided into subgroups, the statistical analyses may be less able to detect educationally 
meaningful program impacts than analyses based on the full sample; and (2) the programmatic features 
occur only after students have enrolled and begun to participate in the coaching program, and features 
may reflect students’ experiences with coaching or other events. Taken together, these make it difficult 

to distinguish in outcomes driven by the features of the coaching experienced by students (variation in 
program impacts), on one hand, and other factors which do not reflect true program impacts, on the other 
hand. This brief highlights results for one student characteristic—underrepresented minority—and one 
feature of coaching—frequency of interactions.° 


* Students may experience coaching differently for multiple reasons—many of which can plausibly be accounted for by the 
matching process used in this study. However, because the matching uses data collected before coaching starts, it is still 
possible that either or both of the following could be related to the features of coaching and the outcomes of interest: a) 
student characteristics that were not captured in the matching (e.g., responsiveness to coaching), or b) students’ experi- 
ences that occur after matching (e.g., success with college coursework). That possibility means it is not possible to causally 
link differences in features of coaching to differences in outcomes. For example, a student may struggle in her first year 
in college, for reasons completely separate from coaching and/or the specific characteristics used in the matching, and 
that student may seek out coaching support more frequently. In this case, more coaching would be related to poor college 
outcomes. On the other hand, a student who is succeeding academically, also for reasons that have nothing to do with 
coaching and characteristics used in the matching, may seek out coaching more frequently to learn about how to do even 
better. In this case, more coaching would be related to more positive outcomes. In both cases, students are self-selecting 
into more frequent coaching yet their outcomes are caused by factors unrelated to the effectiveness of coaching. 


© 


Other student characteristics included as moderators in the full report are: gender, high school GPA, and the type of 
college in which students initially enrolled. The additional features of coaching included in the full report are: duration, 
content focus, and implementation index score. See the full report for a discussion of variation in impacts by these 
moderators. 


12. | SUCCESS BOSTON 


Findings 


The one-on-one transition supports provided by SBC 


improve student outcomes along several important Sum mary of Study Findings 
dimensions on the path to college completion, including 

how long students persist in college, their academic Pag SBC coached students are 11% 
achievement while in college, and their timely comple- - more likely than non-coached 

tion of applications required to access available federal peers to persist into the second year of 
financial aid. Relying on a rigorous, quasi-experimental college, and 21% more likely to persist into 
design—namely local and focal matching—the results the third year? of college than non-coached 
demonstrate that SBC students (the “treatment group” students. 


in the evaluation) have better early college outcomes 


College GPA of SBC coached 


than do their carefully matched peers not participating 
students is 8% higher than that for 


non-coached students, and SBC coached 
students spend 10% more time in good 


in SBC (“comparison group”). 


fir Persistence into the Second academic standing. 
= and Third Years of College y= Coached students are 9% more 


A key premise of the SBC model is that students are likely to renew their FAFSA than 
more likely to persist in college as coaches help them non-coached students. 

navigate and manage the academic, financial, logistical, 
and social-emotional challenges typically faced by 4 Third-year persistence rates are based on the 2013 
beginning college students. To test this hypothesis, BPS graduates only. 

this evaluation uses two primary outcome measures 


to assess persistence: persistence into the second year 
of college and persistence into the third year of college. These two outcomes measure whether students 
who enrolled in college after their high school graduation returned to college in the fall of each successive 
academic year or had already completed a certificate or degree. Persistence into the second year of college 
indicates whether students in the 2013 and 2014 cohorts enrolled in the fall semester of 2014 and 2015, 
respectively, or had completed a degree or credential. Correspondingly, persistence into the third year of 
college reflects whether students in the 2013 cohort enrolled in the 
SBC STUDENTS ARE MORE fall semester of 2015 or had completed a degree or credential (data on 
LIKELY TO PERSIST INTO 2014 graduates’ enrollment in fall 2015 were not available in time for 
THEIR SECOND AND THIRD analyses described in this report). 
YEARS OF COLLEGE THAN 
NON-COACHED STUDENTS. 


SBC students are more likely than students in the comparison group 
to persist into their second and third years of college. The top two 
bars in Exhibit 3 show that 83 percent of SBC students (the treatment 
group) and 75 percent of non-coached students (the comparison group) persisted into the second year 

of college. The 8.1 percentage point impact of SBC on persistence is positive, statistically significant, and 
equivalent to an 11 percent increase in persistence. Both groups persisted into the second year at higher 
rates than the 72 percent national average for students who started college in fall 2014 (NSC 2016). 
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The bottom two bars in Exhibit 3 show that SBC has an impact on persistence into students’ third year of 
college. The impact into the third year is estimated to be 13 percentage points—equivalent to a 21 percent 
increase in persistence (because this outcome is measured only for the 2013 cohort, it should be interpreted 
with caution). Three-quarters (75 percent) of SBC students in the treatment group persisted into the third 
year of college, whereas less than two-thirds (62 percent) of students in the comparison group did.” 


Key to Exhibits 
* Impact is significant at the 5 percent level Persistence outcomes come from NSC data provided 
+ Moderator is significant at the 5 percent level by BPS and MA ESE and college administrative data 
+ Significance determined with the Benjamini-Hochberg Academic and FAFSA outcomes come from college 
correction. administrative data 
EXHIBIT 3 


Impact of SBC on persistence into second and third years of college 


Persistence 
into 2nd Year 
of Colleget 


Persistence 
into 3rd Year 
of College* 


0 20% 40% 60% 80% 100% 


Percentage of Students 


i Adjusted Comparison Group Mean a Treatment Group Mean 


Exhibit 4 displays the impact estimates on second and third year persistence for SBC students compared 
with non-coached students, by underrepresented minority status. SBC has positive impacts on persistence 
into the second and third years for both underrepresented minority students and non-underrepresented 
minority students. While the estimated impacts are larger for not underrepresented students than for 
underrepresented students, the impacts are not statistically different from each other. That is, the results 
suggest that coaching is not more effective at getting White and Asian/Pacific Islander students (i.e., those 
not underrepresented), to persist than it is for Black, Hispanic, Native American and Other/Multiracial 
students (i.e., those underrepresented). 


The impacts on second and third year persistence, as a function of the frequency of coaching interactions 
experienced by SBC students, are shown in Exhibit 5. SBC students who experience higher numbers of 


” Because there are multiple outcomes within the same domain, the study team applied a statistical adjustment called the 
Benjamini-Hochberg correction. The Benjamini-Hochberg correction allows researchers to adjust impact estimates when 
testing multiple outcomes within a single domain; essentially, it accounts for the false discovery rate, that is, it controls 
for the fact that sometimes statistical significance happens by chance. 
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EXHIBIT 4 
Impact of SBC on persistence into second and third years of college, 
by underrepresented minority status 


20% 19.4* 

2 15% 
5 
2 

5 10% 
2 

5% 

0 

Persistence into 2nd Year of College Persistence into 3rd Year of College 
fe Underrepresented eS Not Underrepresented 


coach-student interactions are more likely than comparison students to persist into the second and third 
years of college. These results also suggest that the impact of SBC on persistence into the third year is 
strengthened—that is, larger—for those students who experience more coach-student interactions, as 
they have higher rates of persistence into the third year of college than students who experience fewer 
interactions. 


The positive findings across both persistence outcomes are noteworthy for two reasons. First, these 
corroborate Sum and colleagues’ findings that SBC had a positive and statistically significant effect on 
college persistence (2013 and 2014). Second, the findings are consistent with evaluations of other coaching 
interventions shown to improve students’ likelihood of college persistence." 


EXHIBIT 5 
Impact of SBC on persistence into second and third years of college, 
by frequency of coaching interactions 
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Achievement: Cumulative GPA and Good Academic Standing 


Another key hypothesis of the SBC evaluation is that coaching supports can help students manage various 
academic challenges they may face in college, ranging from difficult coursework to course selection and 
time management. By helping students access available campus supports and cope with stressors related 
to collegiate academic demands, coaching can potentially help students improve their academic achieve- 
ment. SBC implementation data indicate that coach-student interactions during the 2014-15 academic year 
most commonly addressed academic topics (e.g., reviewing course syllabi, course selection and degree 
planning, connecting students to on-campus tutoring services); nearly two-thirds (63 percent) of all 8,685 
coaching interactions included an academic focus (Linkow et al. 2015). 


The analyses described below examine the impact of SBC on two primary outcome measures of academic 
achievement: cumulative grade point average (GPA) and good academic standing. Cumulative GPA is 
based upon students’ most recent semester enrolled and corresponds with the following values: A=4.0, 
B=3.0, C=2.0, D=1.0. Good academic standing reflects whether students maintained semester-specific and 
cumulative GPAs of 2.0 or better, or earned more than 66 percent of credits attempted in a given semester, 
and is represented as a proportion of the number of semesters spent in good academic standing out of the 
maximum of four possible semesters. 


Cumulative GPA 


SBC students are maintaining higher cumulative grade point averages (2.45 vs. 2.26) than comparison 
students. The impact of SBC on cumulative GPA is statistically significant, although the magnitude of the 
impact is fairly modest, about one-fifth of one point (see Exhibit 6). 


EXHIBIT 6 
Impact of SBC on cumulative GPA 


Cumulative GPAT 


0 1 2 3 4 
Grade Point Average (GPA) 


|| Adjusted Comparison Group Mean el Treatment Group Mean 


Exhibit 7 displays the impact estimates on cumulative GPA for SBC students compared with non-coached 
students, by underrepresented minority status. SBC students from traditionally underrepresented 
minority groups have higher cumulative GPAs than comparison students. While SBC students from 
non-underrepresented groups also have higher GPAs than comparison students, that difference is not 
statistically significant. 
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SBC students are more likely than comparison students to achieve higher cumulative GPAs if they experi- 
ence more coach interactions (see Exhibit 8). 


EXHIBIT 7 EXHIBIT 8 
Impact of SBC on cumulative GPA, Impact of SBC on cumulative GPA, 
by underrepresented minority status by frequency of coaching interactions 
.29* 
0.30 0.30 
.26 
0.25 0.25 
0.20 0.20 
5 0.15 5 0.15 
0.10 0.10 
0.05 0.05 
0.00 0.00 


a High Frequency eel Low Frequency 


aa Underrepresented aa Not Underrepresented 


Good Academic Standing 

SBC students are estimated to spend more semesters in good academic standing (78 percent of semesters) 
than are non-coached students in the comparison group (71 percent of semesters). The statistically 
significant 7 percentage point impact on time spent in good academic standing could translate into faster 
pathways to college completion, as students in good standing are not placed on academic probation, which 
too often means that students lose momentum whether by taking subsequent semesters off and/or losing 
financial aid for the next semester. 


EXHIBIT 9 
Impact of SBC on good academic standing 


Good Academic 7* 
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Exhibit 10 displays the impact estimates on good academic standing for SBC students compared with 
non-coached students, by underrepresented minority status. SBC has positive impacts on good academic 
standing for both underrepresented minority students and non-underrepresented minority students. 


SBC students are more likely than comparison students to spend more semesters in good academic stand- 
ing if they experience more coach interactions (see Exhibit 11). The results also suggest that the impacts of 
SBC on semesters spent in good academic standing are stronger for students who experience more rather 
than fewer interactions. 


Over half of the coaching interactions recorded in 2014-2015 covered academic topics. These results suggest 
that this support had an impact on SBC students’ academic success during their first two years in college. 


EXHIBIT 10 EXHIBIT 11 
Impact of SBC on good Impact of SBC on good 
academic standing, by academic standing, by frequency 
underrepresented minority status of coaching interactions 
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FAFSA Renewal 


One SBC goal is to help students navigate the financial aid process, and presumably help reduce 


the gap between the cost of college and what students can actually afford. Consequently, the 
study examines students’ Free Application for Federal Student Aid renewal, to measure whether students 
complete and submit FAFSA renewal applications for their 

second year of college, unlocking access to federal student aid. SBC STUDENTS WHO INTERACT 
MORE FREQUENTLY WITH THEIR 
COACH ARE MORE LIKELY THAN 
NON-SBC STUDENTS TO RENEW 
THEIR FAFSAS 


Exhibit 12 shows that although the majority of both groups of 
students renew their FAFSAs, SBC students in the treatment 
group renew at a higher rate; the difference in renewal rates 
(7 percentage points) is positive, statistically significant, and 
equivalent to a 9 percent increase. Among SBC students, 85 
percent renewed their FAFSA for their second year of college, versus 78 percent of comparison students. 
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That high proportions of SBC and non-coached students in the group renew their FAFSAs for their second 

year in college reflects the high proportion of students eligible for federal financial aid in the study sample; 
specifically, approximately 87 percent of SBC students and 84 percent of comparison students were eligible 

for free or reduced-price lunch in high school. 


EXHIBIT 12 
Impact of SBC on FAFSA renewal 


FAFSA Renewal 7* 
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Exhibit 13 displays the impact estimates of SBC on FAFSA renewal for SBC students compared with non- 
coached students, as a function of underrepresented minority status. SBC students renew their FAFSAs 
at higher rates than comparison students if they are not from a traditionally underrepresented minority 
group. The impact on FAFSA renewal for SBC students from underrepresented groups, while positive, is 
not statistically significant. 


Similar to the results for persistence, cumulative GPA, and good academic standing, SBC students are 
more likely than comparison students to renew their FAFSAs if they experience more coach interactions 
overall (see Exhibit 14). SBC students renew their FAFSAs at a higher rate than comparison students, 
which may make the second year of college more affordable for SBC students. 


EXHIBIT 13 EXHIBIT 14 
Impact of SBC on FAFSA renewal, Impact of SBC on FAFSA renewal, 
by underrepresented minority status by frequency of coach interactions 
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Discussion 


Success Boston Coaching provides multi-faceted resources to 


THE EFFECTS OF SBC CAN BE 


have demonstrably improved outcomes for college-entering CHARACTERIZED AS POSITIVE 
students.“ SBC supports include knowledge of the college, AND LARGE. 
relationship-building, logistics assistance, nudges and remind- 


participating students, including supports that, individually, 


ers to complete time-sensitive tasks, proactive outreach, meeting students where they are, time manage- 
ment skills, helping students become more self-sufficient, and providing social-emotional supports. 


Perhaps uniquely so, SBC integrates these different features into a single intervention that improves 
students’ outcomes across all three outcome domains examined (persistence, achievement, and financial 
aid). 


The magnitude of the SBC impacts estimated in this analysis is generally comparable to those for similar 
programs focused on improving college student outcomes.*” These other studies have observed effects 

on college persistence and achievement equivalent to about 10-15 percent increases over the control group 
means. The estimated impacts of SBC on persistence into the third year" and on FAFSA renewal, however, 
are larger than typically seen in the literature. Overall, then, that means the effects of SBC can be charac- 
terized as positive and large. 


The 2015 SBC implementation report documents the intensity of coaching; it describes how SBC coaches 
provided students with an average of four hours of one-on-one coaching, typically delivered through eight 
30-minute in-person meetings. Additionally, coaches interacted with students via email and text message, 
corresponding to approximately 1.4 interactions a month over the academic year. The impact results 
suggest that the SBC model has meaningful and significant promise. A model in which professional 
coaches deliver sustained, proactive, and responsive support 


within a larger education network may not necessarily need to SBC HAS LARGER IMPACTS ON 
be “high touch” to accomplish the overarching program goal: STUDENTS WHO EXPERIENCE 
to move more students more quickly and effectively along the MORE, COMPARED TO FEWER, 
pathway to college completion. INTERACTIONS. 


Examining variation in impacts as a function of how SBC 

students experience coaching (e.g., frequency of interactions) provides some evidence that the impacts 

of coaching vary in important ways. Specifically, there are statistically significant impacts on students 
who experience more coaching interactions overall. The exploratory analyses also indicate that impacts 
vary across particular features of coaching; SBC has larger impacts on students who experience more, 
compared to fewer, interactions. The frequency of coaching interactions may moderate some of the 
impacts, such that coaching has a stronger effect when students experience 10 or more interactions during 
the year, or at least once a month during a typical academic year. This result can inform decisions about 


™ Recall that this outcome is measured only for the fall 2013 cohort. 
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program implementation—specifically, about how often coaches should interact with students to yield 
positive student outcomes. 


The results reported here demonstrate that SBC is having the desired effects on students from Boston. 

SBC participants are persisting longer, achieving higher academic progress, and taking the necessary 
steps to maintain funding for college—all good signs of progress toward college completion. The study 
will continue to monitor students’ progress over the next several years, culminating in a 2020 report on 
impacts as of five and six years after high school graduation. That report will help address questions about 
whether the positive impacts reported here persist, and whether Success Boston has continued to progress 
toward its central goal: at least 70 percent of BPS college entrants earning college credentials. 
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